1. Field of the Invention
The present invention relates to a method for determining a photographic environment when an object is photographed by an imaging apparatus including an XY-address-scanning imaging device such as a complementary metal oxide semiconductor (CMOS) imaging device. The present invention further relates to an imaging apparatus that determines a photographic environment.
2. Description of the Related Art
When an object is photographed by an imaging apparatus such as a digital video camera or a digital still camera, when particularly motion pictures are taken, it is important whether or not photographing is conducted under fluorescent light. Fluorescent lamps cause flickering, and therefore, measures against fluorescent flickering need be taken.
When an object is photographed by an imaging apparatus under fluorescent lamps that are powered directly by a household AC power supply, a temporal fluctuation of brightness perception, called fluorescent flicker, occurs in the video signal from a photographic output. Fluorescent flicker is caused by the difference between the frequency at which the brightness of fluorescent light changes (or the amount of light changes), which is twice as high as the frequency of the AC power supply, and the vertical sync frequency (or imaging frequency) of the imaging apparatus.
FIG. 1 shows fluorescent flicker when an object is photographed by an NTSC-compatible CCD (Charge Coupled Device) imaging apparatus under non-inverted fluorescent light in a region where the frequency of the power supply is 50 Hz. As shown in FIG. 1, one field is 1/60 second (that is, the vertical sync frequency is 60 Hz), and the period over which the brightness of fluorescent light fluctuates is 1/100 second. Thus, the timing of exposure for each field is deviated relative to the fluctuation of fluorescent-light brightness, and the amount of light exposed on each pixel varies from one field to another (or across fields).
At an exposure time of 1/60 second, the amount of exposure is different in a period a1, a2, or a3 with the same exposure time. At an exposure time shorter than 1/60 second (not equal to 1/100 second, as described below), the amount of exposure is different in a period b1, b2, or b3 with the same exposure time.
The exposure timing is in synchronization with the fluctuation in fluorescent-light brightness every three fields (or 1/20 second). That is, the flicker causes the brightness levels of light to alternately change every three fields. Although the luminance ratios (i.e., perception of flicker) of fields differ depending upon the exposure duration, the flicker intervals are not changed.
A fluorescent lamp radiates white light, and generally includes a plurality of phosphors, e.g., red, green, and blue phosphors each having unique afterglow characteristics. For a period of time from the end of discharge to the beginning of next discharge, which lies in the brightness fluctuation period, each phosphor attenuates illumination according to its afterglow characteristics. Initially, white light is radiated, and the light is attenuated while its hue gradually varies. At this time, if the exposure timing is deviated relative to the fluctuation of the fluorescent-light brightness, not only does the brightness change but also the hue. Moreover, due to the spectral characteristics of fluorescent light, namely, a strong peak is exhibited at a particular wavelength, the fluctuation component of the signal differs depending upon the color.
Such a change in hue and difference in fluctuation component between colors cause so-called color flicker.
If the exposure time is set to an integer multiple of the fluctuation period of the fluorescent-light brightness, i.e., 1/100 second, but not in excess of one field (i.e., 1/60 second), e.g., as shown in the lowermost portion of FIG. 1, if the exposure time is set to 1/100 second, the amount of exposure is constant regardless of the exposure timing. Thus, flicker does not occur. In photographing an object under fluorescent light, instead of setting the shutter speed so that flicker does not occur, it is also conceivable to reduce the amount of flicker caused in the video signal from an imaging output.
In XY-address-scanning imaging apparatuses such as CMOS imaging apparatuses, on the other hand, the pixel exposure timing differs by one period of read clock (pixel clock) in the horizontal and vertical screen directions, and the exposure timing is therefore different on all pixels. Such an XY-address-scanning imaging apparatus causes fluorescent flicker in a different fashion from a CCD imaging apparatus.
Digital video cameras have an NTSC format and a PAL format. Recent digital video cameras are mostly compatible with both NTSC and PAL formats. Such digital video cameras are electrically set to either the NTSC or PAL format depending upon the destination when they are shipped from the factory. Digital still cameras have a progressive format with a frame frequency of 30 Hz.
The frequency of household AC power supply (i.e., fluorescent-lamp driving power supply) is 50 Hz or 60 Hz depending upon the region in Japan or the region or country in the world.
Fluorescent flicker will now be described when photographing is conducted by an NTSC or PAL CMOS imaging apparatus, or a progressive CMOS imaging apparatus having a frame frequency of 30 Hz in a region where the frequency of the power supply is 50 Hz or 60 Hz.
FIG. 2 shows fluorescent flicker when an object is photographed by an NTSC CMOS imaging apparatus under fluorescent light in a region where the frequency of the power supply is 50 Hz.
As described above, in a CMOS imaging apparatus, the exposure timing on pixels is also different in the horizontal screen direction. However, one horizontal period is much shorter than the period over which the brightness of fluorescent light fluctuates, and, actually, it can be presumed that the pixels on the same line are exposed to light at the same timing. The exposure timing for each line in the vertical screen direction is shown in FIG. 2.
As shown in FIG. 2, in the CMOS imaging apparatus, the exposure timing is different from one line to another, as indicated by F0, which indicates a different exposure timing in a particular field. The amount of exposure is also different depending upon the line. Thus, fluctuation in brightness and color is caused by flicker not only across fields but also within a field, and a fringe pattern appears on the screen. On the screen, the fringes themselves lie in the horizontal direction, and the fringes change in the vertical direction.
FIG. 3 is an illustration of screen flicker in a case where an object is formed of uniform patterns. One period (one wavelength) of the fringe pattern is 1/100 second, and 1.666 periods of the fringe pattern are shown on the screen. Let the number of lines read per field be M. One period of the fringe pattern corresponds to L lines read, which is given by L=M* 60/100. Throughout this document and the drawings, the asterisk (*) represents multiplication.
As shown in FIG. 4, five periods (or five wavelengths) of fringe pattern are shown in three fields (or three screens). The fringe pattern looks as if it were vertically advancing when viewed continuously.
Although FIGS. 3 and 4 show only fluctuation in brightness caused by flicker, actually, changes in color also occur, as described above, and the image quality is considerably degraded. In particular, color flicker is noticeable at a high shutter speed (or at a short exposure time). In the CMOS imaging apparatus, color flicker affects the screen, and degradation in image quality becomes more noticeable.
When an object is photographed by an NTSC CMOS imaging apparatus under fluorescent light in a region where the frequency of the power supply is 50 Hz, as shown in FIG. 5A, one field is 1/60 second, and the period over which the brightness of fluorescent light fluctuates is 1/100 second. At either a normal shutter speed where the exposure time is 1/60 second or a high shutter speed where the exposure time is shorter than 1/60 second, as shown in FIG. 5C, continuous flicker on the time axis with intervals of three fields (or three screens) occurs (as if it were vertically advancing when viewed continuously).
If the exposure time is set to an integer multiple of the fluctuation period of the fluorescent-light brightness, i.e., 1/100 second, but not in excess of one field (i.e., 1/60 second), that is, if the exposure time is set to 1/100 second, the amount of exposure is constant regardless of the exposure timing. Thus, flicker, including screen flicker, does not occur.
Flicker for the NTSC CMOS imaging apparatus with a vertical sync frequency of 60 Hz and a power supply frequency of 50 Hz, described above, is shown FIG. 7.
When an object is photographed by an NTSC CMOS imaging apparatus under fluorescent light in a region where the frequency of the power supply is 60 Hz, as shown in FIG. 5B, one field is 1/60 second, and the period over which the brightness of fluorescent light fluctuates is 1/120 second. At a normal shutter speed where the exposure time is 1/60 second, the amount of exposure is constant regardless of the exposure timing, and flicker, including screen flicker, does not occur. At a high shutter speed where the exposure time is shorter than 1/60 second, however, as shown in FIG. 5D, one-field (one-screen) flicker in which flicker in each field (screen) has the same fringe pattern occurs.
If one-screen flicker in which flicker in each screen has the same fringe pattern occurs, a picture (background) component and a flicker component are not distinguished in a video signal sent from the imaging device.
Even at a high shutter speed, if the exposure time is set to the fluctuation period of the fluorescent-light brightness, i.e., 1/120 second, like at a normal shutter speed where the exposure time is 1/60 second, the amount of exposure is constant regardless of the exposure timing. Thus, flicker, including screen flicker, does not occur.
Flicker for the NTSC CMOS imaging apparatus with a vertical sync frequency of 60 Hz and a power supply frequency of 60 Hz, described above, is shown in FIG. 7.
When an object is photographed by a PAL CMOS imaging apparatus under fluorescent light in a region where the frequency of the power supply is 60 Hz, as shown in FIG. 6A, one field is 1/50 second, and the period over which the brightness of fluorescent light fluctuates is 1/120 second. At either a normal shutter speed where the exposure time is 1/50 second or a high shutter speed where the exposure time is shorter than 1/50 second, as shown in FIG. 6C, continuous flicker on the time axis with intervals of five fields (or five screens) occurs (as if it were vertically advancing when viewed continuously).
If the exposure time is set to an integer multiple of the fluctuation period of the fluorescent-light brightness, i.e., 1/120 second, but not in excess of one field (i.e., 1/50 second), that is, if the exposure time is set to 1/120 second or 1/60 second, the amount of exposure is constant regardless of the exposure timing. Thus, flicker, including screen flicker, does not occur.
Flicker for the PAL CMOS imaging apparatus with a vertical sync frequency of 50 Hz and a power supply frequency of 60 Hz, described above, is shown in FIG. 7.
When an object is photographed by a PAL CMOS imaging apparatus under fluorescent light in a region where the frequency of the power supply is 50 Hz, as shown in FIG. 6B, one field is 1/50 second, and the period over which the brightness of fluorescent light fluctuates is 1/100 second. At a normal shutter speed where the exposure time is 1/50 second, the amount of exposure is constant regardless of the timing exposure, and flicker, including screen flicker, does not occur. At a high shutter speed where the exposure time is shorter than 1/50 second, however, as shown in FIG. 6D, one-field (one-screen) flicker in which flicker in each field (each screen) has the same fringe pattern occurs.
Even at a high shutter speed, if the exposure time is set to the fluctuation period of the fluorescent-light brightness, i.e., 1/100 second, like at a normal shutter speed where the exposure time is 1/50 second, the amount of exposure is constant regardless of the exposure timing. Thus, flicker, including screen flicker, does not occur.
Flicker for the PAL CMOS imaging apparatus with a vertical sync frequency of 50 Hz and a power supply frequency of 50 Hz, described above, is shown in FIG. 7.
When an object is photographed by a progressive CMOS imaging apparatus having a frame frequency of 30 Hz under fluorescent light in a region where the frequency of the power supply is 50 Hz, although not shown, one frame is 1/30 second (or the vertical sync frequency is 30 Hz), and the period over which the brightness of fluorescent light fluctuates is 1/100 second. Either at a normal shutter speed where the exposure time is 1/30 second or at a high shutter speed where the exposure time is shorter than 1/30 second, continuous flicker on the time axis with intervals of three frames (or three screens) occurs (as if it were vertically advancing when viewed continuously).
If the exposure time is set to an integer multiple of the fluctuation period of the fluorescent-light brightness, i.e., 1/100 second, but not in excess of one frame (i.e., 1/30 second), that is, if the exposure time is set to 1/100 second, 1/50 second, or 3/100 second, the amount of exposure is constant regardless of the exposure timing. Thus, flicker, including screen flicker, does not occur.
Flicker for the progressive CMOS imaging apparatus with a vertical sync frequency of 30 Hz and a power supply frequency of 50 Hz, described above, is shown in FIG. 7.
When an object is photographed by a progressive CMOS imaging apparatus having a frame frequency of 30 Hz under fluorescent light in a region where the frequency of the power supply is 60 Hz, although not shown, one frame is 1/30 second, and the period over which the brightness of fluorescent light fluctuates is 1/120 second. At a normal shutter speed where the exposure time is 1/30 second, the amount of exposure is constant regardless of the exposure timing, and flicker, including screen flicker, does not occur. At a high shutter speed where the exposure time is shorter than 1/30 second, however, one-frame (one-screen) flicker in which flicker in each frame (each screen) has the same fringe pattern occurs.
Even at a high shutter speed, if the exposure time is set to an integer multiple of the fluctuation period of the fluorescent-light brightness, i.e., 1/120 second, that is, if the exposure time is set to 1/120 second, 1/60 second, or 1/40 second, like at a normal shutter speed where the exposure time is 1/30 second, the amount of exposure is constant regardless of the exposure timing. Thus, flicker, including screen flicker, does not occur.
Flicker for the progressive CMOS imaging apparatus with a vertical sync frequency of 30 Hz and a power supply frequency of 60 Hz, described above, is shown in FIG. 7.
In photographing an object using a CMOS imaging apparatus under fluorescent light, instead of setting the shutter speed so that flicker does not occur, it is also conceivable to reduce the amount of flicker caused in the video signal from an imaging output.
More specifically, when continuous flicker on the time axis with intervals of a plurality of vertical periods (or screens) occurs, shown in FIG. 4 or FIGS. 5A and 5C or 6A and 6C, the continuous flicker is utilized to estimate the flicker component in the video signal of the photographic output, and the video signal of the photographic output is corrected according to the estimation. For example, the gain of the video signal is adjusted according to the estimated flicker component, or the estimated flicker component is subtracted from the video signal. In this way, the amount of flicker component is reduced in the video signal of the photographic-output.
When a still image is taken by a digital video camera or digital still camera capable of taking both motion pictures and still images, all pixels in one screen can be exposed to light at the same timing (more specifically, exposure on all pixels in one screen starts and stops at the same time) even in an XY-address-scanning imaging apparatus such as a CMOS imaging apparatus, and the occurrence of fluorescent flicker is avoided. In this case, unlike taking a motion picture where there are limitations in reading speed, a video signal is slowly read from the imaging device with the shutter mechanically closed to block light.
In an XY-address-scanning imaging apparatus, such as a CMOS imaging apparatus, therefore, in order to set the shutter speed so that flicker does not occur in the video signal from a photographic output or reduce the amount of flicker caused in the video signal, it is necessary to determine whether or not photographing is conducted under fluorescent light, and, if under fluorescent light, it is further necessary to determine whether the frequency of the fluorescent-lamp driving power supply is 50 Hz or 60 Hz.
It is desirable that white balance (WB) adjustment control, auto exposure (AE) adjustment control, etc., be optimized depending upon under fluorescent light or non-fluorescent light. Thus, it is necessary to determine whether or not photographing is conducted under fluorescent light.
One method for determining a photographic environment is disclosed in Japanese Unexamined Patent Application Publication No. 7-336586. In this method, external light is directly measured by a special photometric sensor, and it is determined whether or not a fluctuation component exists in an output signal of the measured light to determine whether or not photographing is conducted under fluorescent light. If it is determined that photographing is conducted under fluorescent light due to the existence of fluctuation component, the frequency of the phosphor-driving power supply is detected by detecting the frequency of the fluctuation component.
The method disclosed in this publication requires a special photometric sensor, and considers connection or communication between the photometric sensor and an imaging apparatus. Therefore, the imaging apparatus must be large and expensive.
Accordingly, demands exist for a method for determining whether or not photographing is conducted under fluorescent light based on the video signal from a photographic output and for detecting the frequency of the fluorescent-lamp driving power supply.
A method disclosed in Japanese Unexamined Patent Application Publication No. 2001-111887 includes:    (1) integrating pixel values in the horizontal screen direction to generate flicker component data with less influence of the background (picture);    (2) averaging the integration data across a plurality of screens to determine an average value corresponding to the background component;    (3) normalizing the integration data using the average value to determine the flicker component data from which influence of the background is removed;    (4) performing a Fourier transform on the flicker component data in the vertical screen direction to extract the spectrum of only the flicker frequency component;    (5) comparing the level of spectrum with a threshold value to determine whether or not flicker occurs, that is, whether or not photographing is conducted under fluorescent light, and determining whether the frequency of the fluorescent-lamp driving power supply is 50 Hz or 60 Hz; and    (6) changing the shutter speed according to the determination to prevent the occurrence of flicker.
A method disclosed in Japanese Unexamined Patent Application Publication No. 2002-84466 includes:    (1) integrating pixel values in the horizontal screen direction to determine an integration value with less influence of the background (picture);    (2) determining a flicker index value from the amount of change in the integration value from frame to frame according to a particular evaluation formula;    (3) determining whether or not flicker occurs, that is, whether or not photographing is conducted under fluorescent light, and determining whether the frequency of the fluorescent-lamp driving power supply is 50 Hz or 60 Hz based on the determined flicker index value; and    (4) changing the shutter speed according to the determination to prevent the occurrence of flicker.
However, the method disclosed in Japanese Unexamined Patent Application Publication No. 2001-111887 or No. 2002-84466 does not overcome the problem of one-screen flicker shown in FIGS. 5B and 5D or 6B and 6D in which flicker in each screen has the same fringe pattern. When such flicker occurs, a picture (background) component and a flicker component are not distinguished in the video signal of the photographic output, and no fluctuation component is detected across fields of the video signal. In this case, it cannot be determined that photographing is conducted under fluorescent light.
In order to solve this problem, Japanese Unexamined Patent Application Publication No. 2002-84466 discloses a method in Embodiment 2, including:    (0) initially, setting the shutter speed to an integer multiple of 1/120 second;    (1) in this setting, integrating pixel values in the horizontal screen direction to determine an integrated value with less influence of the background (picture);    (2) determining a flicker index value from the amount of change in the integrated value from frame to frame according to a particular evaluation formula;    (3) determining whether or not photographing is conducted under fluorescent light with a power supply frequency of 50 Hz based on the determined flicker index value;    (4) if it is determined that photographing is conducted under fluorescent light with a power supply frequency of 50 Hz, setting the shutter speed to an integer multiple of 1/100 second; and    (5) if it is determined that photographing is not conducted under fluorescent light with a power supply frequency of 50 Hz, setting the shutter speed to an integer multiple of 1/120 second, at which flicker does not occur under fluorescent light with a power supply frequency of 60 Hz.
This method also has a problem, however. If photographing is not conducted under fluorescent light with a power supply frequency of 50 Hz, the shutter speed is set to an integer multiple of 1/120 second without determining whether or not photographing is conducted under fluorescent light with a power supply frequency of 60 Hz. Thus, if photographing is not conducted under fluorescent light with a power supply frequency of 50 Hz or 60 Hz, or if photographing is conducted under non-fluorescent light where flicker does not occur, the shutter speed is limited more than necessary. In this method, information about whether or not photographing is conducted under fluorescent light is not finally obtained, although this information is useful for WB adjustment control and AE adjustment control.